Skip to content

Autonomous: fix PM2 startup, FMU scratch and runner connection leaks - #313

Open
tanushchauhan wants to merge 2 commits into
mainfrom
tanush/deploy-hil-jobs
Open

tanushchauhan wants to merge 2 commits into
mainfrom
tanush/deploy-hil-jobs

Conversation

@tanushchauhan

Copy link
Copy Markdown
Member

Replaces #289 after renaming the source branch to tanush/deploy-hil-jobs. The previous PR has the review history.

What

Fix three failures exposed by the first live deployment: PM2's JavaScript wrapper left the dashboard "online" without starting HTTP, Docker's non-executable scratch prevented FMPy from loading the native FMU, and SQLite transaction contexts leaked connections until the worker exhausted file handles.

Launch Node directly in a single PM2 fork, permit native-library loading only in bounded FMU scratch, and close every database connection after commit/rollback. Add regression tests for all three cases.

Record the operator-approved rollout: dashboard under telemetry's existing electric PM2 account, restricted accounts/bridge on HIL, and loopback access through SSH only. No new domain or Tailscale listener was installed. Existing telemetry containers, viewer process and public route are unchanged. ADR-020 records the post-merge account decision; the adjacent runbook records live checks and rollback.

How it's tested

  • python3 -m unittest discover -s apps/job_runner -p 'test_*.py': 46 passing tests, including connection closure on commit and rollback and per-template executable scratch. New regressions fail against the merged code.
  • node --test apps/job_runner/*.test.mjs: 31 passing tests, including native PM2 entrypoint execution, heap bounds and the default, empty and custom config-path cases. The new startup regression fails against the merged template.
  • Ruff, strict MkDocs and git diff --check pass.
  • Live telemetry → restricted SSH → HIL → Docker acceptance: CPU job/results, queued/running cancellation, pause/resume, unchanged container identity through worker restart, dashboard restart recovery, GPU completion, and an FMU profile completing all 12 simulated seconds in 80.563 wall seconds.
  • Live resource soak: 4,000 database operations leave descriptor count at 4; worker stays at 4–6 descriptors over 30 seconds. A further CPU job and archive download pass afterward. Worker service group measured 14.65 MiB / 2.73% of one core; idle dashboard measured 58.13 MiB RSS / 0.0% of one core over a separate 30-second sample.
  • Bazel is unavailable locally; Python tests ran directly. CI will run the Bazel target.

Checklist

  • Behavior changes come with a test that fails without them.
  • No test deleted, skipped or loosened; no lint suppression added.
  • Adjacent READMEs and live deployment record updated.

@coderabbitai

coderabbitai Bot commented Nov 20, 2026

Copy link
Copy Markdown

Important

Review skipped

Ignore keyword(s) in the title.

⛔ Ignored keywords: 1

  • Autonomous:

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@lhre-ultron

lhre-ultron Bot commented Nov 20, 2026

Copy link
Copy Markdown
Contributor

Safe to merge. The three fixes do what the PR says, and I found no real problems in this push.

What I checked

Checked: I read the whole diff.

  • Database connections: connect() is now a @contextmanager that yields the connection inside closing(...) and with db:. Commit and rollback therefore happen before the close, and an exception raised in the body still rolls back and closes. The updated FullDatabase test wrapper matches the new interface.
  • FMU scratch: only the fmu template gets the exec tmpfs. CPU and GPU keep the default non-executable mount, and --read-only, --network none, --cap-drop ALL and no-new-privileges are unchanged.
  • PM2 template: it launches Node directly with interpreter: 'none' in fork mode. The config-path fallback handles both an unset and an empty LHR_JOBS_CONFIG.
  • Docs: the ADR-020 entry, the mkdocs.yml nav entry, the README table and the runbook are consistent with each other and with the code.

Didn't check: I did not run the live HIL, PM2 or Docker acceptance steps, and I did not run Bazel. I also did not run the Node and Python test suites in this pass. The live results rest on the PR description and the runbook.

Assumed: I assumed the connect() callers behave as in #289. They all use with self.connect() as db and only read the rows they fetched inside that block. I assumed PM2 is started from the same Node binary that process.execPath resolves to when the ecosystem file is evaluated.

Ultron reviewed 4a881ef · it re-reviews on new pushes

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant